Segmented Topic Model for Text Classification and Speech Recognition

This paper presents a new segmented topic model (STM) to explore the topic regularities and simultaneously partition the text or spoken documents into coherent segments. The topic model based on the latent Dirichlet allocation (LDA) is adopted to extract the topics and is strengthened by incorporating a Markov chain to detect the segments in a document. STM is trained according to a variational Bayesian procedure where a Viterbi decoder is inherent in carrying out the document segmentation. Each segment is represented by a Markov state, and so the nonstationary stylistic and contextual information are captured. The word variations within a document are compensated. In the experiments, STM outperformed LDA for text classification using ICASSP dataset and for speech recognition using WSJ corpus. 1 Introduct ion Latent Dirichlet allocation (LDA) [4] has been proposed to generalize new documents and adopted for document representation in text classification and summarization systems [5] as well as language model adaptation in speech recognition system [6]. LDA represents the documents based on bag-of-words scheme and ignores the position of words. However, the usage of words is varied in different segments of a document even if it involves the same topic. Such variations affect the correctness of document representation and word prediction. In this study, a Markov chain is merged in LDA to detect the stylistically-similar segments and estimate the time-varying word statistics of a document. Each segment indicates a specific writing or spoken style in composition of a text or spoken document. This segmented topic model (STM) exploits the topic information across documents and the word variations within a document. In STM parameter inference, a Viterbi variational inference algorithm is presented by running a Viterbi decoding stage in a variational Bayesian EM (VB-EM) procedure. The proposed STM is evaluated for text classification and speech recognition. 2 Segmented topic model The word distributions in different paragraphs of a text or spoken document are varied due to the composition style and document structure. In addition to topic information, the temporal positions of words are embedded in natural language, e.g. scientific articles and broadcast news documents. Particularly, when a scientific article is related to a specific scope of research topics, the word distributions are varied in the segments of abstract, introduction and experiment. To compensate the temporal variations, a Markov chain is merged to characterize the dynamics of words in different segments as displayed by the graphical representation of STM in Figure 1. The K-dimensional topic mixture vector θ is drawn from a Dirichlet distribution with parameter . α The topic sequence z is generated by a multinomial distribution with parameter . θ The state sequence } , , { 1 N s s L = s is generated by a Markov chain with an initial state parameter π and a S S × state transition probability matrix }. { 1 n n s s a − = A Each word is associated with a topic and a segment. The marginal likelihood of a document over unseen variables , θ z and s is yielded by θ A π θ B α θ A π B α w d s s p z p s z w p p p N